Sublinear Time Approximation of Text Similarity Matrices

We study algorithms for approximating pairwise similarity matrices that arise in natural language processing. Generally, computing a matrix n data points requires Omega(n^2) computations. This quadratic scaling is significant bottleneck, especially when similarities are computed via expensive functions, e.g., transformer models. Approximation methods reduce this complexity, often by using small subset of exactly to approximate the remainder complete matrix. Significant work focuses on efficient approximation positive semidefinite (PSD) matrices, which kernel methods. However, much less understood about indefinite (non-PSD) NLP. Motivated observation many these still somewhat close PSD, we introduce generalization popular Nystrom method setting. Our algorithm can be applied any and runs sublinear time size matrix, producing rank-s with just O(ns) show our method, along simple variant CUR decomposition, performs very well variety arising NLP tasks. demonstrate high accuracy approximated tasks document classification, sentence similarity, cross-document coreference.